a

GAO

United States General Accounting Office

Report to Congressional Requesters

October 2002 NURSING HOMES

Public Reporting of 

Quality Indicators Has 

Merit, but National 

Implementation Is 

Premature

GAO-03-187
CMS’s initiative to augment existing public data on nursing home quality has

considerable merit, but its planned November 2002 implementation does not

allow sufficient time to ensure the indicators it publishes are appropriate

and useful to consumers. CMS’s plan urges consumers to consider nursing

homes with positive quality indicator scores, in effect, attempting to use

market forces to encourage nursing homes to improve the quality of care.

However, CMS is moving forward without adequately resolving a number of

important open issues on the appropriateness of the indicators chosen for

national reporting or the accuracy of the underlying data.

To develop and help select the quality indicators, CMS hired two

organizations with expertise in health care data—Abt Associates, Inc. and

the National Quality Forum (NQF). Abt identified a list of potential quality

indicators and tested them to verify that they represented the actual quality

of care individual nursing homes provide. Although the full Abt report on

validation of the indicators was not available as of October 28, 2002,

GAO’s review of the available portions of the report raised serious questions

about the basis for moving forward with national reporting at this time.

NQF, which was created to develop and implement a national strategy for

measuring health care quality, was hired to review Abt’s work and identify

core indicators for national reporting. To allow sufficient time to review

Abt’s validation report, NQF agreed to delay its recommendations for

national reporting until 2003. CMS limited its own evaluation of its six-state

pilot program for the initiative so that the November 2002 implementation

date could be met. Early results were expected in October 2002, leaving

little time to incorporate them into the national rollout. Despite the lack of a

final report from NQF and an incomplete pilot evaluation, CMS has

announced a set of indicators it will begin reporting nationally in November

2002.

GAO has serious concerns about the potential for public confusion by the

quality information published, especially if there are significant changes to

the quality indicators due to the NQF’s review. CMS’s proposed reporting

format implies a precision in the data that is lacking at this time. While

acknowledging this problem, CMS said it prefers to wait until after the

national rollout to modify the presentation of the data. GAO’s analysis of

data currently available from the pilot states demonstrated there was ample

opportunity for the public to be confused, highlighting the need for clear

descriptions of the data’s limitations and easy access to impartial experts

hired by CMS to operate consumer hotlines. CMS has not yet demonstrated

its readiness to meet these consumer needs either directly or through the

hotlines fielding public questions about confusing or conflicting quality data.

CMS acknowledged that further work is needed to refine its initiative, but

believes that its indicators are sufficiently valid, reliable, and accurate to

move forward with national implementation in November 2002 as planned.

NURSING HOMES

Public Reporting of Quality Indicators

Has Merit, but National Implementation Is

Premature

www.gao.gov/cgi-bin/getrpt?GAO-03-187.

To view the full report, including the scope

and methodology, click on the link above.

For more information, contact Kathryn G.

Allen at (202) 512-7118.

Highlights of GAO-03-187, a report to

congressional requesters

October 2002

GAO was asked to review the

Centers for Medicare & Medicaid

Services (CMS) initiative to

publicly report additional

information on its “Nursing Home

Compare” Web site that is intended

to help consumers choose a

nursing home. GAO examined

CMS’s development of the new

nursing home quality indicators

and efforts to verify the underlying

data used to calculate them. GAO

also reviewed the assistance CMS

offered the public in interpreting

and comparing indicators available

in its six-state pilot program,

launched in April 2002, and its own

evaluation of the pilot. The new

indicators are scheduled to be used

nationally beginning in November

2002.

GAO is recommending that the

Administrator of CMS delay the

national reporting of quality

indicators to allow sufficient time

to resolve important issues

regarding appropriate indicators

for public reporting and to

implement a program to review the

accuracy of the data on which the

indicators are based. During this

time, CMS also should more

thoroughly evaluate the results of

its six-state pilot to assess how

information is presented and to

improve assistance available

through consumer hotlines.
Page i GAO-03-187 Nursing Home Quality Indicators

Contents

Letter 1

Results in Brief 3

Background 4

Appropriateness of Quality Indicators Proposed for Public 

Reporting Is Unresolved 7

CMS Has Not Addressed Concerns About the Underlying Accuracy 

of MDS Data Used to Develop Quality Indicators 10

The Public May Be Confused by Quality Data and CMS Is Not 

Prepared to Respond to Consumers’ Questions 15

Pilot Evaluation Is Limited and Will Not Be Completed Prior to 

National Reporting of Quality Indicators 20

Conclusions 21

Recommendations for Executive Action 22

Comments from CMS and the NQF and Our Evaluation 22

Appendixes

Appendix I: Comparison of Quality Indicators Proposed by NQF and CMS 

for National Rollout 27

Appendix II: Comments from the Centers for Medicare & Medicaid 

Services 28

Appendix III: Comments from the National Quality Forum 38

Appendix IV: GAO Contact and Staff Acknowledgments 40

Tables Table 1: Error Rates in MDS Items Used to Develop Selected 

Quality Indicators 14

Table 2: Percentage of Nursing Homes in the Six Pilot States with 

Missing Quality Indicator Scores 17

Table 3: Comparison of Publicly Reported Nursing Home Quality 

Indicator Scores and Quality-of-Care Survey Deficiencies 

in Six Pilot States 18

Figure Figure 1: Error Rates in MDS Assessments in 30 Nursing Homes 

Reviewed by Abt 13
Contents

Page ii GAO-03-187 Nursing Home Quality Indicators

Abbreviations

CMS Centers for Medicare & Medicaid Services

HCFA Health Care Financing Administration

HHS Department of Health and Human Services

MDS minimum data set

NQF National Quality Forum

OIG Office of Inspector General

QIO Quality Improvement Organization
Page 1 GAO-03-187 Nursing Home Quality Indicators

United States General Accounting Office

Washington, D.C. 20548

A

October 31, 2002 Letr

The Honorable Charles E. Grassley

Ranking Minority Member

Committee on Finance

United State Senate

The Honorable Christopher S. Bond

United State Senate

Almost half of all Americans over the age of 65 will spend time in a nursing 

home at some point in their lives. A series of congressional hearings since 

1998 has focused considerable attention on the unacceptably high number 

of nursing homes with repeated, serious care problems that harmed 

residents or placed them at risk of death or serious injury. Given the large 

number of nursing home residents and the growing public concerns over 

quality-of-care problems, the Centers for Medicare & Medicaid Services 

(CMS) has shown a strong commitment to providing assistance to 

individuals and their families in choosing a nursing home. CMS is the 

agency within the Department of Health and Human Services (HHS) that 

manages Medicare and Medicaid and oversees compliance with federal 

nursing home quality standards.1

 In 1998, the agency launched a Web site—

“Nursing Home Compare”—that has progressively expanded the 

availability of public information on nursing homes and the quality of care 

provided. 2 Initially, it posted data on deficiencies identified during routine 

state nursing home inspections, known as surveys. Data were later added 

on resident characteristics, such as the percentage of residents with 

pressure sores or physical restraints, nursing staff levels, and deficiencies 

found during state investigations of complaints. 

In November 2001, CMS announced a 12-month timeline for an initiative to 

(1) augment existing public data on nursing home quality, and (2) provide 

assistance to nursing homes to help improve their quality of care. In 

addition to the valuable data already available on its Web site, CMS 

proposed including newly developed quality indicators that permit a fairer 

comparison across homes by adjusting for differences in residents’ health 

1

In June 2001, the agency’s name was changed from the Health Care Financing 

Administration (HCFA) to CMS. In this report, we continue to refer to HCFA where our 

findings apply to the organizational structure and operations associated with that name. 

2

www.Medicare.gov/NHCompare/home.asp.
Page 2 GAO-03-187 Nursing Home Quality Indicators

characteristics. Quality indicators are essentially numeric warning signs of 

problems, such as more frequent than expected pressure sores among 

nursing home residents. They are based on data from facility-reported 

assessments—known as the minimum data set (MDS)—conducted at 

established intervals during each resident’s nursing home stay. The 

initiative also envisioned a new role for Medicare Quality Improvement 

Organizations (QIO): engaging in partnership building and local 

promotional activities designed to put quality information into the hands of 

consumers and working with nursing homes on a voluntary basis to help 

improve quality of care.3

 In April 2002, CMS launched a six-state pilot to 

refine the initiative before planned nationwide implementation in 

November 2002. The six pilot states are Colorado, Florida, Maryland, Ohio, 

Rhode Island, and Washington. Medicare QIOs are working with from 6 to 

11 nursing homes in each pilot state on projects including improving pain 

management and preventing pressure sores. 

In view of the importance of CMS’s quality indicator initiative and the 

relatively short pilot time frame prior to national implementation, you 

asked us to review the (1) development of the new quality indicators for 

public reporting, (2) status of CMS’s efforts to ensure the accuracy of the 

underlying data used to calculate the quality indicators, (3) assistance 

offered by CMS to the public in understanding the new quality indicator 

data, and (4) results of CMS’s evaluation of the pilot. To do so, we reviewed 

pertinent documents from CMS on the development of the new quality 

indicators, the approaches identified to adjust for differences in residents’ 

characteristics in each facility, the validation of the new indicators, the 

operation and evaluation of the pilot, and the role of the QIOs. We 

discussed these areas with CMS officials and with researchers from Abt 

Associates, Inc., the lead CMS contractor responsible for the development 

and validation of the new quality indicators. We also interviewed and 

reviewed materials provided by officials of the National Quality Forum 

(NQF), a group CMS contracted with to review Abt’s work and provide 

recommendations on quality indicators for public reporting. 4 We examined 

the consistency of both the quality indicator data available in the six pilot 

3

Under contract with CMS, 37 QIOs (formerly known as Peer Review Organizations) are 

responsible for determining the quality, effectiveness, and efficiency of health care services 

provided to Medicare beneficiaries in all 50 states and the District of Columbia. 

4

NQF is a nonprofit organization created to develop and implement a national strategy for 

health care quality measurement and reporting. NQF has broad participation from 

government and private entities as well as all sectors of the healthcare industry.
Page 3 GAO-03-187 Nursing Home Quality Indicators

states and the extent of agreement between such data and the results of 

nursing home surveys. To determine how CMS is assisting consumers in 

understanding the quality indicator data available in the six pilot states, we 

posed questions about discrepancies we identified between the indicators 

and survey deficiencies to staff who field public inquiries received by the 

Medicare and QIO toll-free telephone numbers. We conducted our work 

from March through September 2002 in accordance with generally 

accepted government auditing standards. 

Results in Brief Overall, CMS’s initiative to augment existing public data on nursing home 

quality has considerable merit but its plan for nationwide implementation 

in November 2002 is premature. Conceptually, CMS’s plan encourages 

consumers making a decision about a nursing home to consider those with 

positive quality indicator scores—a use of market forces to encourage 

poorly performing homes to improve quality of care or face the loss of 

revenue. Such a plan hinges, in part, on appropriate quality indicators that 

consistently distinguish between good and poor care provided by nursing 

homes. However, CMS has not yet adequately resolved a number of open 

issues regarding the appropriateness of the quality indicators selected for 

public reporting and the accuracy of the underlying data. 

CMS contracted with two expert organizations, Abt and NQF, to develop 

and help select quality indicators appropriate for public reporting, but it 

does not intend to await NQF input before proceeding. In August 2002, 

CMS announced a set of quality indicators it intends to begin reporting 

nationally in November 2002. Its selection was based on the results of Abt’s 

efforts to validate the nursing home quality indicators it had developed for 

CMS. Although the full Abt validation report was not available to us as of 

October 28, 2002, our review of the available portions of the report raised 

serious questions about the basis for moving forward with national 

reporting at this time. Moreover, Abt’s finding that the underlying MDS data 

are accurate is not convincing in light of the results of earlier studies that 

identified widespread errors in the accuracy of facility-specific 

assessments used to calculate some of the quality indicators CMS has 

selected for November reporting. In 2001, CMS also contracted with the 

NQF to review Abt’s work and recommend a set of quality indicators for 

national reporting but in June 2002 asked the NQF to delay finalizing its 

recommendations until 2003. The delay will allow NQF to consider both the 

full Abt validation report and the results of an evaluation of the six-state 

pilot established to refine the initiative but will not allow CMS to consider 

NQF’s input before its planned nationwide implementation. 
Page 4 GAO-03-187 Nursing Home Quality Indicators

In addition to concerns over the appropriateness of the quality indicators, 

we found that CMS was not well prepared to respond to consumer 

questions about the quality data and had not allotted sufficient time to 

incorporate lessons learned from its six-state pilot. CMS’s reporting of 

quality indicators in the pilot states was neither consumer friendly nor 

presented in a format consistent with the data’s limitations. For example, 

reporting homes’ actual quality indicator scores rather than rankings—

whether a home was in the bottom, middle, or top range of homes on a 

particular score—could make it difficult for consumers to interpret the 

differences in homes’ scores and could imply a precision that does not 

currently exist. Our analysis of the data also demonstrated the potential for 

public confusion over (1) contradictory information from the quality 

indicators themselves—almost one-fifth of homes in pilot states had an 

equal number of highly positive and highly negative quality indicator scores 

and (2) inconsistencies between quality indicator scores and data on 

deficiencies identified during nursing home surveys—almost one-fifth of 

homes with four or more highly positive quality indicator scores and no 

highly negative scores had at least one serious quality of care deficiency on 

a recent state survey. Moreover, our telephone calls to the Medicare and 

QIO toll-free numbers revealed that CMS was not adequately prepared to 

address questions raised by the public and we received erroneous or 

misleading information in the majority of calls we placed. Finally, CMS’s 

evaluation of the pilot itself is limited and will not be fully completed until 

sometime in 2003—months after the planned national implementation of 

the initiative.

We are recommending that the Administrator of CMS delay the national 

reporting of quality indicators until (1) there is greater assurance that 

quality indicators are appropriate and based on accurate data and (2) a 

more thorough evaluation of the pilot is completed. In commenting on a 

draft of this report, CMS reiterated its commitment to continually improve 

the quality indicators and to work to resolve the issues discussed in our 

report. However, CMS does not intend to delay its initiative before 

resolving the issues raised. CMS believes that its quality measurement 

information is sufficiently reliable, valid, accurate, and useful to move 

forward with national implementation in November 2002 as planned. 

Background Since 1998, the results from state surveys of nursing homes have been the 

principal source of public information on nursing home quality, which is 

posted and routinely updated on CMS’s Nursing Home Compare Web site. 

Under contract with CMS, states are required to conduct periodic surveys 
Page 5 GAO-03-187 Nursing Home Quality Indicators

that focus on determining whether care and services meet the assessed 

needs of the residents and whether homes are in compliance with federal 

quality requirements, such as preventing avoidable pressure sores, weight 

loss, or accidents.5

 During a survey, a team that includes registered nurses 

spends several days at a home reviewing the quality of care provided to a 

sample of residents. States are also required to investigate complaints 

lodged against nursing homes by residents, families, and others. In contrast 

to surveys, complaint investigations generally target a single area in 

response to a complaint filed against a home. Any deficiencies identified 

during routine surveys or complaint investigations are classified according 

to the number of residents potentially or actually affected (isolated, 

pattern, or widespread) and their severity (potential for minimal harm, 

potential for more than minimal harm, actual harm, and immediate 

jeopardy).

To improve the rigor of the survey process, HCFA contracted for the 

development of quality indicators and required their use by state surveyors 

beginning in 1999.6

 Quality indicators are derived from data collected 

during nursing homes’ assessments of residents, known as the minimum 

data set (MDS). The MDS contains individual assessment items covering 17 

areas, such as mood and behavior, physical functioning, and skin 

conditions. MDS assessments of each resident are conducted in the first 14 

days after admission and periodically thereafter and are used to develop a

5

Surveys must be conducted at each home on average every 12 months and no less than 

once every 15 months.

6

The quality indicators used in nursing home surveys were developed by the University of 

Wisconsin under a HCFA-funded contract. See Center for Health Systems Research and 

Analysis, Facility Guide for the Nursing Home Quality Indicators (Madison: University of 

Wisconsin-Madison: September 1999). Surveyors use the indicators to help select a 

preliminary sample of residents and preview information on the care provided to a home’s 

residents prior to the on-site inspection. Prior to their introduction in 1999, selection of the 

sample relied on a listing of residents and their conditions maintained at the nursing home 

and on observation of residents made during a walk through of the facility. As a result of the 

quality indicators, the sample selection is more systematic and surveyors are better 

prepared to identify potential care problems. However, the quality indicators used during 

surveys were not developed for public reporting because they were viewed as providing an 

indication of a potential quality problem that required validation through an on-site survey. 
Page 6 GAO-03-187 Nursing Home Quality Indicators

resident’s plan of care.7 Facility-reported MDS data are used by state 

surveyors to help identify quality problems at nursing homes and by CMS to 

determine the level of nursing home payments for Medicare; some states 

also use MDS data to calculate Medicaid nursing home payments.

Because it also envisioned using indicators to communicate nursing home 

quality to consumers, HCFA recognized that any publicly reported 

indicators must pass a very rigorous standard for validity and reliability. 

Valid quality indicators that distinguish between good and poor care 

provided by nursing homes would be a useful adjunct to existing quality 

data. Such indicators must also be reliable—that is, they must consistently 

distinguish between good and bad care. HCFA contracted with Abt to 

review existing quality indicators and determine if they were suitable for 

public reporting. Abt catalogued and evaluated 143 existing quality 

indicators, including those used by state surveyors. It also identified the 

need for additional indicators both for individuals with chronic conditions 

who are long-term residents of a facility and for individuals who enter a 

nursing home for a short period, such as after a hospitalization (a postacute 

stay). According to Abt, a main concern about publicly reporting quality 

indicators was that the quality indicator scores might be influenced by 

other factors, such as residents’ health status. Abt concluded that the 

specification of appropriate risk adjustment models was a key requirement 

for the validity of any quality indicators. Risk adjustment is important 

because it provides consumers with an “apples-to-apples” comparison of 

nursing homes by taking into consideration the characteristics of individual 

residents and adjusting quality indicator scores accordingly. For example, a 

home with a disproportionate number of residents who are bedfast or who 

present a challenge for maintaining an adequate level of nutrition—factors 

that contribute to the development of pressures sores—may have a higher 

pressure sore score. Adjusting a home’s quality indicator score to fairly 

represent to what extent a home does—or does not—admit such residents 

is important for consumers who may wish to compare one home to 

another. After several years of work, Abt recommended 39 risk-adjusted 

quality indicators to CMS in October 2001. Twenty-two were based on 

existing indicators and the remaining 17 were newly developed by Abt, 

including 9 indicators for nursing home residents with chronic conditions 

7

MDS assessments are conducted for all nursing home residents within 14 days of admission 

and at quarterly and yearly intervals unless there is a significant change in condition. 

Medicare beneficiaries in a Medicare-covered stay are assessed on or before the 5th, 14th, 

30th, 60th, and 90th day of their stays to determine if their Medicare coverage should continue.
Page 7 GAO-03-187 Nursing Home Quality Indicators

and 8 indicators for individuals who enter a nursing home for a short 

period. 

In September 2001, CMS contracted with the NQF to review Abt’s work 

with the objective of (1) recommending a set of quality indicators for use in 

its planned six-state pilot and (2) developing a core set of indicators for 

national implementation of the initiative scheduled for late 2002. NQF 

established a steering committee to accomplish these two tasks.8

 The 

steering committee met in November 2001 and identified 11 indicators for 

use in the pilot, 9 of which were selected by CMS. The committee made its 

selection from among Abt’s list of 39 indicators but it did not recommend 

use of Abt’s risk-adjustment approach. Moreover, the steering committee 

indicated that it would not be limited to the same Abt list in developing its 

recommended core set of indicators for national implementation. In April 

2002, NQF released a draft consensus report identifying the indicators it 

had distributed to its members and the public for comment on their 

potential inclusion in the national implementation. 9 Under its contract, 

NQF was scheduled to make a final recommendation to CMS prior to the 

national reporting of quality indicators. 

Appropriateness of 

Quality Indicators 

Proposed for Public 

Reporting Is 

Unresolved 

CMS’s initiative to augment existing public data on nursing home quality 

has considerable merit but more time is needed to assure that the 

indicators proposed by CMS for public reporting are appropriate in terms 

of their validity and reliability. Based on work by Abt to validate the 

indicators it developed for CMS, the agency selected quality indicators for 

national reporting. The full Abt validation report—which is important for a 

thorough analysis of the appropriateness of the quality indicators--was still 

not available to us as of October 28, 2002. Our review of available portions 

of the Abt report, however, raised serious questions about whether testing 

and validation of the selected indicators has been sufficient to move 

8

The steering committee consists of 12 stakeholders representing health researchers, 

geriatricians, state survey agencies, state Medicaid directors, health systems, and others.

9

The NQF relies on a consensus process led by a steering committee that initially conducts 

an overall assessment in a particular area and gathers input from NQF members, 

nonmembers, and expert advisory panels. The steering committee then recommends a set of 

draft measures, indicators, or practices for review. Next, the draft recommendations are 

distributed for review and comment—first to NQF members and then to the general public. 

Following this open review period, a revised product is distributed to NQF members for a 

vote. The NQF Board of Directors must ultimately approve matters under consideration 

before the consensus process is complete.
Page 8 GAO-03-187 Nursing Home Quality Indicators

forward with national reporting at this time. Moreover, CMS plans to 

initiate national reporting before it receives recommendations from NQF, 

its contractor, on appropriate quality indicators. 

On August 9, 2002, CMS announced the 10 indicators selected for its 

nationwide reporting of quality indicators, which it plans to launch in midNovember 2002. CMS selected these indicators from those that Abt had 

validated in its August 2, 2002, validation report.10 Abt classified the 

indicators it studied as to the degree of validity—top, middle, or not valid. 

The indicators that CMS selected were in the top category with one 

exception—residents in physical restraints—which was in the middle 

category. The objective of Abt’s validation study was to confirm that the 

indicators reflect the actual quality of care that individual nursing facilities 

provide, after taking into account resident and facility-level characteristics. 

For example, a validation analysis could confirm that a low percentage of 

pressure sores among residents was linked to a facility’s use of procedures 

to prevent their development. Successful validation reduces the chance 

that publicly reported data could misrepresent a high-quality facility as a 

low-quality facility—or vice versa. 

CMS’s decision to implement national reporting in November 2002 is 

troubling, given the issues raised by our review of the available portions of 

Abt’s validation report. Although we asked CMS for a copy of Abt’s 11 

technical appendixes, as of October 28, 2002, they were still undergoing 

review and were not available to us. The technical appendixes are essential 

to adequately understand and evaluate Abt’s validation approach. Our 

review of the available portions of the Abt report raised serious questions 

about whether the effort to date has been sufficient to validate the 

indicators. The validation study is based on a sample that is drawn from six 

states; it is not representative of nursing homes nationwide and may not be 

representative of facilities in these six states. Selected facilities were 

allowed to decline participation and about 50 percent did so. For those 

facilities in the validation study, Abt deemed most of the indicators as 

valid—that is, better care processes were associated with higher quality 

indicator scores, taking into account resident and facility-level 

characteristics. However, we could not evaluate these findings because Abt 

provided little information on the specific care processes against which the 

10Abt Associates, Inc., HRCA Research and Training Institute, and Brown University, 

Validation of Long-Term and Post-Acute Care Quality Indicators, final draft report 

prepared for CMS, Office of Clinical Standards and Quality, Aug. 2, 2002.
Page 9 GAO-03-187 Nursing Home Quality Indicators

indicators were validated. Unresolved questions also exist about the risk 

adjustment of the quality indicators. Risk adjustment is a particularly 

important element in determining certain quality indicators because it may 

change the ranking of individual facilities—a facility that is among the 

highest on a particular quality indicator without risk adjustment may fall to 

the middle or below after risk adjustment—and vice versa. Data released 

by CMS in March 2002 demonstrated that Abt’s risk adjustment approaches 

could either lower or raise facility scores by 40 percent or more. Although 

such changes in ranking may be appropriate, Abt did not provide detailed 

information on how its risk adjustment approaches changed facility 

rankings or a basis for assessing the appropriateness of the changes. 

In addition to the questions raised by our review of the Abt validation 

report, CMS is not planning to wait for the expert advice it sought on 

quality indicators through its contract with the NQF. Under this contract, 

the NQF steering committee issued a consensus draft in April 2002 with a 

set of potential indicators for public reporting. The steering committee had 

planned to complete its review of these indicators using its consensus 

process by August 2002.11 In late June, however, CMS asked NQF to delay 

finalizing its recommendations until early 2003 to allow (1) consideration 

of Abt’s August 2002 report on the validity of its indicators and riskadjustment methods—including the technical appendices, when they 

become available and (2) a review of the pilot evaluation results expected 

in October 2002. An NQF official told us that the organization agreed to the 

delay because the proposed rapid implementation timeline had been a 

concern since the initiative’s inception. CMS’s list of quality indicators for 

the November 2002 national rollout did not include six indicators under 

consideration by NQF—depression, incontinence, catheterization, bedfast 

residents, weight loss, and rehospitalization (see app. I). Instead, CMS 

intends to consider NQF’s recommendations and revise the indicators used 

in the mid-November national rollout sometime next year. 

11NQF indicated that it viewed its list as a starting point for a stronger, more robust set of 

future indicators. Because nursing homes include both medical care and social services, 

NQF believes that a core set of indicators should cover several other highly important areas 

in addition to clinical quality of care, including resident quality of life; measures of resident 

and family satisfaction; and the nursing home environment, such as food quality and number 

of residents per room. A March 2002 report prepared for CMS acknowledged that clinical 

indicators are less important to the public than issues such as facility cleanliness and a 

caring staff. See Barents Group of KPMG Consulting, Inc., Nursing Home Consumer Choice 

Campaign Needs Assessment Report (McLean, Va.: Mar. 14, 2002). 
Page 10 GAO-03-187 Nursing Home Quality Indicators

CMS is also moving forward without a consensus on risk adjustment of 

quality indicators. CMS is planning to report one indicator with facilitylevel adjustment based on a profile of residents’ status at admission, and 

two indicators both with and without this Abt-developed risk adjuster.12 

However, both Abt and NQF have concluded that adjusting for the type of 

residents admitted to the nursing home required further research to 

determine its validity.13 We believe that reporting the same indicator with 

and without facility-level risk adjustment could serve to confuse rather 

than help consumers. Two of the three consultants hired by NQF 

specifically recommended against the use of facility-level adjustments in 

public reporting at this time. We also found that, as of October 1, 2002, CMS 

had not reached internal consensus on how to describe the riskadjustment methods used in each of the 10 indicators it plans to begin 

reporting nationally in November 2002. Several agency officials agreed with 

our assessment that the descriptions on its Web site were inconsistent with 

Abt’s own descriptions of the risk adjustment associated with each 

indicator. 

CMS Has Not 

Addressed Concerns 

About the Underlying 

Accuracy of MDS Data 

Used to Develop 

Quality Indicators

Two different Abt studies have presented CMS with conflicting messages 

about the accuracy of MDS data. Abt’s August 2002 quality indicator 

validation report suggested that the underlying data used to calculate most 

indicators were, in the aggregate, very reliable. However, our analysis of 

more detailed facility-level data in a February 2001 Abt report raised 

questions about the reliability of some of the same MDS data. Because MDS 

data are used by CMS and some states to determine the level of nursing 

home payments for Medicare and Medicaid and to calculate quality 

indicators, ensuring its accuracy at the facility level is critical both for 

determining appropriate payments and for public reporting of the quality 

indicators. Recognizing the importance of accurate MDS data, CMS is in the 

process of implementing a national MDS accuracy review program 

12CMS explained that its decision to use facility-level adjustments was influenced by “great 

stakeholder interest” in how this new risk-adjustment methodology affected them.

13NQF based it recommendation on the work of a Special Advisory Panel of three 

independent consultants who were asked to assist in resolving concerns about the technical 

complexity of Abt’s risk adjustment approaches, particularly its proposed facility-level 

adjustments. Specifically, the April 2002 NQF consensus draft recommended priority 

funding for (1) research regarding the selection of appropriate risk factors; (2) comparisons 

of the different risk adjustment methodologies for nursing home performance data, as 

applied to each quality indicator; and (3) validation of different risk-adjustment methods.
Page 11 GAO-03-187 Nursing Home Quality Indicators

expected to become fully operational in 2003, after the nationwide 

reporting of quality indicators begins in November 2002. We recently 

reported that CMS’s review program is too limited in scope to provide 

adequate confidence in the accuracy of MDS assessments in the vast bulk 

of nursing homes nationwide.14 

Abt’s August 2, 2002, validation report concluded that the reliability of the 

underlying MDS data used to calculate 39 quality indicators ranged from 

acceptable to superior, with the data for only 1 indicator proving 

unacceptable.15 Abt’s findings were based on a comparison of assessments 

conducted by its own nurses to assessments performed by the nursing 

home staff in 209 sample facilities. For each quality indicator, Abt reported 

the overall reliability for all of the facilities in its sample.16 However, 

because quality indicators will be reported for each nursing home, overall 

reliability is not a sufficient assurance that the underlying MDS data are 

reliable for each nursing home. Although Abt did not provide information 

on MDS reliability for individual facilities, it noted that reliability varied 

considerably within and across states.

Earlier work by Abt and others calls into question the reliability of MDS 

data. Abt’s February 2001 report on MDS data accuracy identified 

significant variation in the rate of MDS errors across the 30 facilities 

sampled. 17 Differences between assessments conducted by Abt’s nurses 

and the nursing home staff were classified as errors by Abt. Error rates for 

all MDS items averaged 11.7 percent but varied across facilities by a factor 

of almost two—from 7.8 percent to 14.5 percent. As shown in figure 1, the 

14See U.S. General Accounting Office, Nursing Homes: Federal Efforts to Monitor Resident 

Assessment Data Should Complement State Activities, GAO-02-279 (Washington, D.C.: Feb. 

15, 2002).

15Validation analysis was incomplete for two additional indicators.

16As noted earlier, Abt’s sample may not be representative as only 50 percent of homes 

agreed to participate. 

17Abt Associates, Inc., Development and Testing of a Minimum Data Set Accuracy 

Verification Protocol, final report prepared for HCFA, Feb. 27, 2001. The authors of this 

study computed the combined error rate for individual facilities by weighting the error rates 

for Medicare and non-Medicare assessments using the proportion of Medicare (.32) to nonMedicare (.68) assessment items for the entire sample of 30 facilities. However, the 

proportion of Medicare to non-Medicare assessments varied across facilities. For example, 

in one facility there were more Medicare than non-Medicare assessments. We therefore 

recomputed facility error rates using the proportion of Medicare to non-Medicare MDS 

assessment items for each facility.
Page 12 GAO-03-187 Nursing Home Quality Indicators

majority of error rates were higher than 10.5 percent. Furthermore, error 

rates for some of the individual MDS items used to calculate the quality 

indicators were much higher than the average error rate.18 According to 

Abt, the least accurate sections of the MDS included physical functioning 

and skin conditions. Abt also noted that there was a tendency for facilities 

to underreport residents with pain.19 MDS items from these portions of the 

assessment are used to calculate several quality indicators that CMS plans 

to report nationally in November 2002—activities of daily living, pressure 

sores, and pain management. Table 1 shows that the error rate across the 

residents sampled ranged from 18 percent for pressure sores to 42 percent 

for pain intensity.20 Abt’s February 2001 findings were consistent with 

areas that states have identified as having a high potential for error, 

including activities of daily living and skin conditions.21 Moreover, a study 

by the HHS Office of Inspector General (OIG), which identified differences 

between the MDS assessment and the medical record, found that activities 

of daily living was among the areas that provided the greatest source of 

differences.22 In addition, the OIG report noted that 40 percent of the 

nursing home MDS coordinators it surveyed identified the physical 

functioning section, used to calculate the quality indicator on activities of 

daily living, as the most difficult to complete. Some coordinators explained 

that facility staff view a resident’s capabilities differently and thus the 

assessments tend to be subjective. 

18Abt did not report error rates for individual items at the facility level.

19More recently, state survey agency officials in three pilot states told us that they are 

concerned that the public reporting of quality indicators may lead to underreporting of 

certain problem areas, such as pain management.

20Abt did not provide error rates for individual items that are adjusted to reflect the extent of 

the differences in assessments conducted by Abt and the facility nurses. 

21GAO-02-279, pp. 16-18.

22OIG used the term “differences” rather than errors because its methodology did not permit 

a specific determination as to why the differences occurred. See HHS Office of Inspector 

General, Nursing Home Resident Assessment: Quality of Care, OEI-02-99-00040 

(Washington, D.C.: December 2000). In commenting on this report, CMS expressed 

reservations about the OIG’s methodology and interpretation of CMS documents used to 

perform the study. The OIG had recommended that nursing homes be required to establish 

an “audit trail” to document support for certain MDS elements. CMS disagreed, noting that it 

did not expect all information in the MDS to be duplicated elsewhere in the medical record. 

We concur with the OIG’s position that, given the use of MDS data in adjusting nursing home 

payments and producing quality indicators, documenting the basis for the MDS assessments 

in the medical record is critical to assessing their accuracy. See GAO-02-279.
Page 13 GAO-03-187 Nursing Home Quality Indicators

Figure 1: Error Rates in MDS Assessments in 30 Nursing Homes Reviewed by Abt

Source: GAO analysis of data from Abt Associates, Inc., Minimum Data Set Accuracy.

0

1

2

3

4

5

6

7

8

Percent of assessments with errors

Number of nursing homes

<=8.5

8.6-9.5

9.6-10.5

11.6-12.5

12.6-13.5

10.6-

13.6-14.5

11.5
Page 14 GAO-03-187 Nursing Home Quality Indicators

Table 1: Error Rates in MDS Items Used to Develop Selected Quality Indicators

Source: Abt Associates, Inc., Minimum Data Set Accuracy.

As part of CMS’s efforts to improve MDS accuracy, its contractor is still 

field-testing the on-site aspect of its approach, which is not expected to be 

implemented until 2003.23 Although Abt’s February 2001 report found 

widespread MDS errors, CMS intends to review roughly 1 percent of the 

MDS assessments prepared over the course of a year, which numbered 14.7 

million in 2001. Moreover, only 10 percent of the reviews will be conducted 

on-site at nursing homes. In contrast, our prior work on MDS found that 9 

of the 10 states with MDS-based Medicaid payment systems that examine 

MDS data’s accuracy conduct periodic on-site reviews in all or a significant 

portion of their nursing homes, generally examining from 10 to 40 percent 

of assessments. On-site reviews heighten facility staff awareness of the 

importance of MDS data and can lead to the correction of practices that 

contribute to MDS errors. We reported earlier that CMS’s approach may 

yield some broad sense of the accuracy of MDS assessments on an 

aggregate level but is insufficient to provide confidence about the accuracy 

of MDS assessments in the vast bulk of nursing homes nationwide. 

MDS item Error rate (percent)

Physical functioning: used to calculate quality indicator on decline in activities of 

daily living

Bed mobility 39

Transfer 34

Eating 37

Toilet use 35

Skin condition: used to calculate quality indicator on prevalence of pressure sores

Pressure sore 18

Health condition: used to calculate quality indicator on inadequate pain 

management

Pain frequency 39

Pain intensity 42

23On-site reviews focus on determining whether a resident’s medical record supports the 

MDS assessment completed by the facility. If the MDS assessment is recent, the review may 

also include direct observation of the resident and interviews with nursing home staff who 

have recently evaluated or treated the resident. Off-site reviews of MDS data include 

examining trends in assessments across facilities to identify aberrant or inconsistent 

patterns. 
Page 15 GAO-03-187 Nursing Home Quality Indicators

The Public May Be 

Confused by Quality 

Data and CMS Is Not 

Prepared to Respond 

to Consumers’ 

Questions

While CMS is strongly committed to making more information available to 

the public on nursing home quality and such an initiative has considerable 

merit, the agency had not demonstrated a readiness to assist the public in 

understanding and using those data. We found that CMS’s reporting of 

quality indicators in the six pilot states was neither consumer friendly nor 

reported in a format consistent with the data’s limitations, implying a 

greater degree of precision than is currently warranted. Our analysis of the 

data currently available in the six pilot states demonstrated the potential 

for public confusion over both the quality indicators themselves and 

inconsistencies with other available data on deficiencies identified during 

nursing home surveys—which, to date, are the primary source of public 

data on nursing home quality. Moreover, our phone calls to the Medicare 

and QIO toll-free numbers revealed that CMS was not adequately prepared 

to address consumers’ questions raised by discrepancies between 

conflicting sources of quality data. 

Our review of the quality indicators on the CMS Web site found that the 

presentation of the data was not consumer friendly and that the reporting 

format implies a greater confidence in the data’s precision than may be 

warranted at this time. Quality indicators are reported as the percentage of 

residents in a facility having the particular characteristics measured by 

each indicator. The Web site explains that having a low percentage of 

residents with pressure sores or pain is better than having a high 

percentage. In the six-state pilot, the public can compare a nursing home’s 

score to the statewide and overall average for each quality indicator. We 

believe that equating a high score with poor performance is 

counterintuitive and could prove confusing to consumers.24 Despite the 

Web site’s explanation of how to interpret the scores, the public might well 

assume that a high score is a positive sign.

In addition, reporting actual quality indicator scores rather than the range 

of scores a home falls into for an indicator—a low, medium, or high score—

can be confusing and implies a confidence in the precision of the results 

24Stakeholders that commented on NQF’s April 2002 draft set of indicators suggested that 

the quality indicator scores be reported as the percentage of residents not having the 

particular characteristic measured by each indicator—e.g., reporting that 80 percent of 

residents were not restrained rather than reporting that 20 percent of residents were

restrained. If quality indicators were reported this way, having a high score would be better 

than having a low score. CMS indicated that it had received similar comments but will not 

make any changes prior to the national rollout in November 2002. 
Page 16 GAO-03-187 Nursing Home Quality Indicators

that is currently a goal rather than a reality. Consumers will find it difficult 

to assess a home with a score that is 5 to 10 percentage points from the 

state average. Such a home could be an outlier—one of the best or the 

worst on that indicator; alternatively, it could be that the home was close to 

the state average because the outliers involved much larger differences. 

Concerns about the validity of the indicators and the potential reliability of 

the data make comparisons of homes with similar scores questionable. 

Consumers may be misled if a difference of several percentage points 

between two homes is perceived as demonstrating that one is better or 

worse than the other. To partially address these types of concerns, 

Maryland has reported quality indicator data on its own Web site since 

August 2001 in ranges rather than individual values. Thus, it indicates if a 

facility falls into the bottom 10 percent, the middle 70 percent, or the top 20 

percent of facilities in the state.

Consumers may also be confused about how to interpret missing 

information. Although the CMS Web site explains that quality indicator 

scores are not reported for nursing homes with too few residents, it does 

not acknowledge the extent of such missing data. We found that 6 percent 

of all nursing homes in the six pilot states have no score for any of the nine 

quality indicators and that, for individual indicators, from 9 percent to 40 

percent of facilities have missing scores (see table 2).25 When data for 

homes of potential interest to consumers are not reported, consumers may 

need some assistance in how to incorporate such instances into their 

decisionmaking. 

25Chronic-care quality indicator scores were not reported for nursing homes with fewer than 

30 residents after excluding some residents, e.g., those with certain clinical characteristics 

or those with missing data necessary to calculate a score. Short-stay quality indicator scores 

were not reported for nursing homes with fewer than 20 residents after excluding some 

residents.
Page 17 GAO-03-187 Nursing Home Quality Indicators

Table 2: Percentage of Nursing Homes in the Six Pilot States with Missing Quality 

Indicator Scores

Source: GAO analysis of CMS quality indicator data available on its Web site for the six pilot states. 

Consumer confusion may also occur when quality indicator scores send 

conflicting messages about the overall quality of care at a home. We found 

that the Web site data for a significant number of facilities contained such 

inconsistencies. Seventeen percent of nursing homes in the six pilot states 

had an equal number of highly positive and highly negative quality indicator 

scores. We defined highly positive scores as those indicating that a facility 

was among the 25 percent of homes with the lowest percentage of 

residents exhibiting poor outcomes, such as a decline in their ability to 

walk or use the toilet. In contrast, facilities with a highly negative score 

were among the top 25 percent of homes with poor outcomes. We also 

found that 37 percent of nursing homes with four or more highly positive 

quality indicator scores had two or more highly negative scores.26

In addition, our comparison of survey deficiency data available on the Web 

site with quality indicator scores also revealed inconsistencies. For 

example, 17 percent of nursing homes with four or more highly positive 

quality indicator scores and no highly negative scores—seemingly “good” 

nursing homes—had at least one serious quality-of-care deficiency on a 

Quality indicators

Percentage of nursing homes with

missing score

Chronic-care quality indicators

Decline in activities of daily living 19

Infections 16

Inadequate pain management 16

Pressure sores 16

Physical restraints used daily 9

Weight loss 21

Short-stay quality indicators

Failure to improve and manage delirium 36

Inadequate pain management 35

Improvement in walking 40

26The largest number of highly positive or highly negative scores that any nursing home in 

the pilot states had was seven.
Page 18 GAO-03-187 Nursing Home Quality Indicators

recent state survey. We have found that serious deficiencies cited by state 

nursing home surveyors were generally warranted and indeed reflected 

instances of documented actual harm to nursing home residents.27 

Moreover, 73 percent of nursing homes with four or more highly negative 

quality indicator scores—seemingly “bad” facilities—had no serious 

quality-of-care deficiencies on a recent survey (see table 3). The latter 

situation is consistent with our past work that surveyors often miss serious 

quality-of-care problems.28 Nevertheless, consumers will generally lack 

such insights on the reliability of state surveys that would permit them to 

better assess the available data on quality of care. 

Table 3: Comparison of Publicly Reported Nursing Home Quality Indicator Scores 

and Quality-of-Care Survey Deficiencies in Six Pilot States

Note: A serious quality-of-care deficiency indicates that surveyors found actual harm to residents.

Source: GAO analysis of quality indicator and survey deficiency data available on CMS’s Web site for 

the six pilot states.

With the apparent need for assistance to consumers in interpreting and 

using this information, the important role of the Medicare and QIO toll-free 

numbers is evident. We requested and reviewed copies of the Medicare 

hotline and QIO scripts and found that they did not address the issue of 

27See U.S. General Accounting Office, Nursing Homes: Proposal to Enhance Oversight of 

Poorly Performing Homes Has Merit, GAO/HEHS-99-157 (Washington, D.C.: June 30, 1999).

28See U.S. General Accounting Office, California Nursing Homes: Care Problems Persist 

Despite Federal and State Oversight, GAO/HEHS-98-202 (Washington, D.C.: July 27, 1998). 

State

Nursing homes with four or more

highly positive quality indicator

scores and no highly negative

quality indicator scores that had at

least one serious quality-of-care

survey deficiency (percent)

Nursing homes with four

or more highly negative

quality indicator scores

that had no serious

quality-of-care survey

deficiency (percent)

Colorado 0 75

Florida 11 81

Maryland 29 78

Ohio 27 69

Rhode Island 0 92

Washington 25 47

Total 17 73
Page 19 GAO-03-187 Nursing Home Quality Indicators

responding to questions about conflicting or confusing quality data. 

Furthermore, our calls to the Medicare hotline and to QIO toll-free 

numbers in the six pilot states demonstrated that the staff were not 

adequately prepared to handle basic questions about the quality data 

available under the pilot. CMS officials had told us that Medicare hotline 

callers with complicated questions would be seamlessly transferred to a 

QIO without having to hang up and call another number. Although we 

asked the Medicare hotline staff if another organization might be better 

able to respond to our questions, no one offered to refer us to QIOs, even 

when we specifically asked about them. In fact, one hotline staff member 

told us that a QIO would not be an appropriate referral. Consequently, we 

independently attempted to call the QIOs in the six pilot states. We found 

that it was difficult to reach a QIO staff member qualified to answer 

questions. Each QIO had a toll-free number but neither the automated 

recordings at four QIOs nor operators at the remaining two indicated that 

the caller had reached a QIO.29 In addition, the automated recordings did 

not contain a menu choice for questions about nursing home quality 

indicators.30 We were unable to contact one QIO because the hotline had 

neither an operator nor a voice mail capability. On other calls, after 

reaching a QIO staff person, it frequently took several referrals to identify 

an appropriate contact point. One QIO took 5 working days for a staff 

member to call us back. Four of the five QIOs we contacted explained that 

their primary role was to work with nursing homes to improve quality of 

care. In general, QIO staff were not prepared to respond to consumer 

questions. 

Staff at the Medicare hotline and the QIOs varied greatly in their basic 

understanding of quality indicators and survey deficiencies. While two of 

the nine staff we contacted were generally knowledgeable about different 

types of quality data, others were unable to answer simple questions and 

the majority provided erroneous or misleading data. One QIO staff member 

told us that MDS data were not representative of all residents of a nursing 

home but only presented a “little picture” based on a few residents. 

However, assessments of all residents are taken into consideration in 

calculating quality indicators. When we expressed concern about a home 

identified on the Web site with a “level-3” deficiency, a Medicare hotline 

29The hotline identified each of the six QIOs we called by its proprietary name—not by the 

term QIO or Quality Improvement Organization. For example, the QIO for Ohio is known as 

KePRO, while the QIO for Alaska, Idaho, and Washington is called Qualis Health. 

30A few QIOs did have a menu option for calls about the “nursing home project.” 
Page 20 GAO-03-187 Nursing Home Quality Indicators

staff member incorrectly told us that it was not a serious deficiency 

because level 3 indicated potential harm.31 CMS designates actual harm 

deficiencies as “level-3” deficiencies. A QIO staff member incorrectly told 

us that actual harm pressure sore deficiencies had nothing to do with 

patient care and might be related to paperwork. Our review of survey 

reports has shown that actual harm deficiencies generally involved serious 

quality-of-care problems resulting in resident harm.32 Generally, hotline 

staff did not express a preference for using either nursing home surveys or 

quality indicators in choosing a nursing home. Two QIO staff, however, 

stated that the nursing home survey information gave a better picture of 

nursing home care than the quality indicators, which they judged to be 

imprecise and subject to variability. 

Pilot Evaluation Is 

Limited and Will Not 

Be Completed Prior to 

National Reporting of 

Quality Indicators

CMS’s evaluation of the pilot is limited and will not be completed prior to 

national reporting of quality indicators because of the short period of time 

between the launch of the pilot and the planned November 2002 national 

implementation. According to CMS officials, the pilot evaluation was never 

intended to help decide whether the initiative should be implemented 

nationally or to measure the impact on nursing home quality. While CMS is 

interested in whether nursing home quality actually improves as a result of 

the initiative, it will be some time before such a determination can be made. 

Thus, CMS focused the pilot evaluation on identifying improvements that 

could be incorporated into the initiative’s design prior to the scheduled 

national implementation in November 2002. A CMS official told us that 

initial pilot evaluation results were expected by early October 2002, 

allowing just over a month to incorporate any lessons learned. In 

commenting on a draft of this report, CMS stated that it was using 

preliminary findings to steer national implementation.33 The final results of 

the pilot evaluation will not be completed until sometime in 2003.

31CMS identifies nursing home deficiencies on its Nursing Home Compare Web site using 

numbers with 2 equivalent to potential for more than minimal harm and 3 equivalent to 

actual harm. 

32U.S. General Accounting Office, Nursing Homes: Proposal to Enhance Oversight of 

Poorly Performing Homes Has Merit, GAO/HEHS-99-157 (Washington D.C.: June 30, 1999).

33CMS also plans to incorporate information from a contractor’s study completed prior to 

the pilot to determine how it could better motivate consumers to use nursing home quality 

information to make better informed decisions. See Barents Group, Nursing Home 

Consumer Choice Campaign Needs Assessment Report.
Page 21 GAO-03-187 Nursing Home Quality Indicators

CMS’s evaluation of the pilot is focused on identifying how to communicate 

more effectively with consumers about the initiative and how to improve 

QIO interaction with nursing homes. Specifically, CMS will assess whether 

(1) the target audiences were reached; (2) the initiative increased 

consumer use of nursing home quality information;34 (3) consumers used 

the new information to choose a nursing home; (4) QIO activities 

influenced nursing home quality improvement activities; (5) nursing 

homes found the assistance provided by QIOs useful; and (6) the initiative 

influenced those who might assist consumers in selecting a nursing home, 

such as hospital discharge planners and physicians. Information is being 

collected by conducting consumer focus groups, tracking Web site “hits” 

and toll-free telephone inquiries, administering a Web site satisfaction 

survey, and surveying nursing homes, hospital discharge planners, and 

physicians. As of late August 2002, CMS teams were also in the process of 

completing site visits to stakeholders in the six pilot states, including QIOs, 

nursing homes, ombudsmen, survey agencies, nursing home industry 

representatives, and consumer advocacy groups. The teams’ objective is to 

obtain a first-hand perspective of how the initiative is working with the goal 

of implementing necessary changes and better supporting the program in 

the future.

Conclusions Although CMS’s initiative to publicly report nursing home quality 

indicators is a commendable and worthwhile goal, we believe that it is 

important for CMS to wait for and consider input from NQF and make 

necessary adjustments to the initiative based on its input. We believe 

several factors demonstrate that CMS’s planned national reporting of 

quality indicators in November 2002 is premature. Our review of the 

available portions of Abt’s validation report raised serious questions about 

whether the effort to date has been sufficient to validate the quality 

indicators. NQF was asked to delay recommending a set of indicators for 

national reporting until 2003, in part, to provide sufficient time for it to 

review Abt’s report. Although limited in scope, CMS’s planned MDS 

accuracy review program will not begin on-site accuracy reviews of the 

data underlying quality indicators until 2003. CMS’s own evaluation of the 

pilot, designed to help refine the initiative, was limited to fit CMS’s 

timetable for the initiative and the preliminary finding were not available 

34In April and May 2002, the number of Web site “hits” for states in the pilot increased 

substantially during the week the pilot was announced and subsequently decreased, but 

they remained higher than before the launch of the pilot.
Page 22 GAO-03-187 Nursing Home Quality Indicators

until October 2002, leaving little time to incorporate the results into the 

planned national rollout. Other aspects of the evaluation will not be 

available until early 2003. We also have serious concerns about the 

potential for public confusion over quality data, highlighting the need for 

clear descriptions of the data’s limitations and easy access to informed 

experts at both the Medicare and QIO hotlines. CMS has not yet 

demonstrated its readiness to meet these consumer needs either directly or 

through the QIOs.

Recommendations for 

Executive Action

To ensure that publicly reported quality indicator data accurately reflect 

the status of quality in nursing homes and fairly compare homes to one 

another, we recommend that the Administrator of CMS delay the 

implementation of nationwide reporting of quality indicators until

• there is greater assurance that the quality indicators are appropriate for 

public reporting—including the validity of the indicators selected and 

the use of an appropriate risk-adjustment methodology—based on input 

from the NQF and other experts and, if necessary, additional analysis 

and testing; and

• a more thorough evaluation of the pilot is completed to help improve the 

initiative’s effectiveness, including an assessment of the presentation of 

information on the Web site and the resources needed to assist 

consumers’ use of the information. 

Comments from CMS 

and the NQF and Our 

Evaluation

CMS and the NQF reviewed and provided comments on a draft of this 

report. (See app. II and app. III, respectively). CMS reiterated its 

commitment to continually improve the quality indicators and to work to 

resolve the issues discussed in our report. Although CMS stated it would 

use our report to help improve the initiative over time, it intends to move 

forward with national implementation in November 2002 as planned. It 

stated that “waiting for more reliability, more validity, more accuracy, and 

more usefulness will delay needed public accountability, and deprive 

consumers, clinicians, and providers of important information they can use 

now.” The NQF commented that it unequivocally supports CMS’s plans to 

publicly report quality indicators but indicated that the initiative would 

benefit from a short-term postponement of 3 to 4 months to achieve a 

consensus on a set of indicators and to provide additional time to prepare 

the public on how to use and interpret the data. We continue to support the 
Page 23 GAO-03-187 Nursing Home Quality Indicators

concept of reporting quality indicators, but remain concerned that a flawed 

implementation could seriously undercut support for and the potential 

effectiveness of this very worthwhile initiative. CMS’s comments and our 

evaluation focused largely on two issues: (1) the selection and validity of 

quality indicators, and (2) lessons learned from CMS’s evaluation of the 

pilot initiative. 

Selection and Validity of 

Quality Indicators

CMS asserts that the quality indicators it plans to report nationally are 

reliable, valid, accurate, and useful and that it has received input from a 

number of sources in selecting the indicators for this initiative. However, 

CMS provided no new evidence addressing our findings regarding the 

appropriateness of the quality indicators selected for public reporting and 

the accuracy of the underlying data. We continue to believe that, prior to 

nationwide implementation, CMS should resolve these open issues. 

CMS intends to move forward with nationwide implementation without a 

requested NQF assessment of the full Abt validation report and without 

NQF’s final recommendations on quality indicators. CMS would not share 

the technical appendices to Abt’s validation report with us because they 

were undergoing review and revision. The technical appendices are critical 

to assessing Abt’s validation approach. CMS’s comments did not address 

our specific findings on the available portions of Abt’s validation report, 

including: (1) the validation results are not representative of nursing 

homes nationwide because of limitations in the selection of a sample of 

nursing homes to participate in the validation study, and (2) Abt provided 

little information on the specific care processes against which the 

indicators were validated or how its risk adjustment approaches changed 

facility rankings and the appropriateness of the changes. Although both Abt 

and the NQF concluded that Abt’s facility-level risk adjustment approach 

required further research to determine its validity, CMS plans to report two 

indicators with and without facility-level adjustments. CMS’s comments 

indicated that it has chosen to report these measures both ways in order to 

evaluate their usefulness and to allow facilities and consumers the 

additional information. We continue to believe that reporting data of 

uncertain validity is inappropriate and, as such, will likely not be useful to 

either facilities or consumers. 

For quality indicators to be reliable, the underlying MDS data used to 

calculate the indicators must be accurate. CMS’s comments did not 

specifically address the conflicting findings on MDS accuracy from Abt’s 

August 2002 validation report and its February 2001 report to CMS. Abt’s 
Page 24 GAO-03-187 Nursing Home Quality Indicators

August 2002 validation report concluded that, in aggregate, the underlying 

MDS data were very reliable but that the reliability varied considerably 

within and across states. Aggregate reliability, however, is insufficient 

because quality indicators are reported separately for each facility. In its 

February 2001 report to CMS, Abt identified widespread errors in the 

accuracy of facility-specific assessments used to calculate some of the 

quality indicators that CMS has selected for reporting in November. CMS 

indicated that its efforts since 1999 have improved MDS accuracy. But 

because CMS does not plan to begin limited on-site MDS accuracy reviews 

until 2003, there is little evidence to support this assertion.

Lessons Learned from 

CMS’s Evaluation of the 

Pilot Initiative

CMS commented that findings from a number of activities evaluating the 

six-state pilot were not available prior to the time we asked for comments 

on our draft report. While final reports are not yet available for some of 

these studies, CMS stated that the pilot allowed it to work through 

important issues and incorporate lessons learned before a national launch. 

We pointed out that the pilot evaluation was limited and incomplete—an 

additional reason to delay the initiative. CMS also did not evaluate a key 

implementation issue—the adequacy of assistance available to consumers 

through its toll-free telephone hotlines. Moreover, the lack of formal 

evaluation reports to help guide the development of a consensus about key 

issues, such as how quality indicators should be reported, is troubling.

In its comments, CMS stated that it was committed to working aggressively 

to help the public understand nursing home quality information using 

lessons learned from the pilot. However, CMS learned about the flaws in its 

hotline operations not from its pilot evaluation but from our attempts to 

use the Medicare and QIO toll-free phone numbers to obtain information on 

quality data. Acknowledging the weaknesses we identified, the agency laid 

out a series of actions intended to strengthen the hotlines’ ability to 

respond to public inquiries, such as providing additional training to 

customer service representatives prior to the national launch of the 

initiative. CMS outlined other steps it plans to take such as providing its 

customer service representatives with new scripts and questions and 

answers to the most frequently asked questions. At the outset of the pilot in 

April 2002, CMS described seamless transfers from the Medicare to the QIO 

hotlines for complicated consumer questions but now acknowledges that 

limitations in QIO telephone technology prevent such transfers. Instead of 

automatic transfers, CMS stated that, when referrals to QIOs are necessary, 

callers will be provided with a direct toll-free phone number. CMS also 

commented that consumers should be encouraged to consider multiple 
Page 25 GAO-03-187 Nursing Home Quality Indicators

types of information on nursing home quality. While we agree, we believe it 

is critical that customer service representatives have a clear understanding 

of the strengths and limitations of different types of data to properly inform 

consumers when they inquire. 

CMS commented that we offered no explanation of the analysis that led us 

to conclude that (1) consumers could be confused because scores on 

quality indicators can conflict with each other and the results of routine 

nursing home surveys, and (2) the public may confuse a high quality 

indicator score with a positive result. Our draft clearly states that our 

findings were based on our analysis of the quality indicator data and survey 

results available in the six pilot states—a database that CMS provided at 

our request. In its comments, CMS provided limited data to support its 

assertion that consumers are not confused by the quality indicators and are 

very satisfied with the current presentation on its Web site. According to 

CMS, over two-thirds of respondents to its August 2002 online satisfaction 

survey of randomly chosen users of Nursing Home Compare information 

said they were highly satisfied with the information, for example, it was 

clearly displayed, easy to understand, and valuable. It is not clear, however, 

that these responses were representative of all nursing home consumers 

accessing the Web site, as CMS implied. For example, CMS informed us 

that this survey was part of a larger survey of all Medicare Web site users, 

which had a low overall response rate of 29 percent. Moreover, of the 654 

respondents to the Nursing Home Compare component of the survey, fewer 

than half (40 percent) were identified as Medicare beneficiaries, family 

members, or friends. NQF feedback to CMS on its Web site presentation 

was consistent with our findings. In commenting on our draft report, NQF 

noted that it had offered informal guidance to CMS, such as using positive 

or neutral wording to describe indicators, exploring alternative ways of 

presenting information about differences among facilities, and ensuring 

that the presentation of the data reflects meaningful differences in topics 

important to consumers. While justifying its current presentation of quality 

indicator data, CMS commented that it is seriously considering not 

reporting individual nursing home scores but rather grouping homes into 

ranges such as the bottom 10 percent, middle 70 percent, and top 20 

percent of facilities in a state. Such a change, however, would not come 

before the national rollout. We agree with CMS that, when grouping homes 

into ranges, homes on the margin—close to the bottom 10 percent or top 20 

percent—may not be significantly different from one another. However, the 

same is true of reporting individual facility scores. Moreover, reporting 

ranges more clearly identifies homes that are outliers for consumers. 
Page 26 GAO-03-187 Nursing Home Quality Indicators

Additional CMS Comments CMS also commented on our characterization of the scope of the nursing 

home quality initiative. CMS stated that we had narrowly framed the 

initiative as one designed solely for consumers, ignoring the QIO’s quality 

improvement activities with individual nursing homes requesting 

assistance. Our report acknowledged and briefly outlined the quality 

improvement role of the QIOs. However, based on our requestors’ concerns 

about the relatively short pilot timeframe prior to national implementation 

of public reporting of quality indicators, we focused our work on that key 

aspect of the initiative. CMS cited its Interim Report on Evaluation 

Activities for the Nursing Home Quality Initiative to support its 

conclusion that the initiative was successful in promoting quality 

improvement activities among nursing homes. The improvements cited in 

the Interim Report were self-reported by facilities and CMS offered no 

insights on the nature of the quality improvement changes. The Interim 

Report was not available when we sent our draft report to CMS for 

comment.

CMS provided several technical comments which we incorporated as 

appropriate. 

As agreed with your offices, unless you publicly announce its contents 

earlier, we plan no further distribution of this report until 30 days after its 

issue date. At that time, we will send copies to the Administrator of CMS, 

appropriate congressional committees, and other interested parties. We 

will also make copies available to others upon request. In addition, the 

report will be available at no charge on the GAO Web site at 

http://www.gao.gov.

If you or your staff have any questions, please call me at (202) 512-7118 or 

Walter Ochinko at (202) 512-7157. GAO staff acknowledgments are listed in 

appendix IV. 

Kathryn G. Allen

Director, Health Care—Medicaid

and Private Health Insurance Issues
Page 27 GAO-03-187 Nursing Home Quality Indicators

Appendix I

Comparison of Quality Indicators Proposed by Apendxies

NQF and CMS for National Rollout ApendxiI

a

This indicator is listed twice because CMS plans to report it with and without facility-level adjustment. 

Source: NQF and CMS.

Indicator

Draft NQF indicators

for national reporting

CMS indicators for

the national rollout

Chronic care (long-stay resident) quality indicators

Decline in activities of daily living a a

Pressure ulcers, high and low risk a

Pressure ulcers a

Pressure ulcersa a

Inadequate pain management a a

Physical restraints used daily a a

Infections a

Weight loss a

Depression without therapy a

Incontinence a

Catheterization a

Bedfast residents a

Postacute (short-stay resident) quality indicators

Failure to improve and manage delirium a a

Failure to improve and manage 

deliriuma

a

Inadequate pain management a a

Improvement in walking a a

Rehospitalizations a
Page 28 GAO-03-187 Nursing Home Quality Indicators

Appendix II

Comments from the Centers for Medicare & 

Medicaid Services ApendxiI
Appendix II

Comments from the Centers for Medicare & 

Medicaid Services

Page 29 GAO-03-187 Nursing Home Quality Indicators
Appendix II

Comments from the Centers for Medicare & 

Medicaid Services

Page 30 GAO-03-187 Nursing Home Quality Indicators
Appendix II

Comments from the Centers for Medicare & 

Medicaid Services

Page 31 GAO-03-187 Nursing Home Quality Indicators
Appendix II

Comments from the Centers for Medicare & 

Medicaid Services

Page 32 GAO-03-187 Nursing Home Quality Indicators
Appendix II

Comments from the Centers for Medicare & 

Medicaid Services

Page 33 GAO-03-187 Nursing Home Quality Indicators
Appendix II

Comments from the Centers for Medicare & 

Medicaid Services

Page 34 GAO-03-187 Nursing Home Quality Indicators
Appendix II

Comments from the Centers for Medicare & 

Medicaid Services

Page 35 GAO-03-187 Nursing Home Quality Indicators
Appendix II

Comments from the Centers for Medicare & 

Medicaid Services

Page 36 GAO-03-187 Nursing Home Quality Indicators
Appendix II

Comments from the Centers for Medicare & 

Medicaid Services

Page 37 GAO-03-187 Nursing Home Quality Indicators
Page 38 GAO-03-187 Nursing Home Quality Indicators

Appendix III

Comments from the National Quality Forum ApendxiI
Appendix III

Comments from the National Quality Forum

Page 39 GAO-03-187 Nursing Home Quality Indicators
Page 40 GAO-03-187 Nursing Home Quality Indicators

Appendix IV

GAO Contact and Staff Acknowledgments ApendxiIV

GAO Contact Walter Ochinko, (202) 512-7157

Acknowledgments The following staff made important contributions to this report: Laura 

Sutton Elsberg, Patricia A. Jones, Dean Mohs, Dae Park, Jonathan Ratner, 

Peter Schmidt, Paul M. Thomas, and Phyllis Thorburn.

(290232)
GAO’s Mission The General Accounting Office, the investigative arm of Congress, exists to 

support Congress in meeting its constitutional responsibilities and to help improve 

the performance and accountability of the federal government for the American 

people. GAO examines the use of public funds; evaluates federal programs and 

policies; and provides analyses, recommendations, and other assistance to help 

Congress make informed oversight, policy, and funding decisions. GAO’s 

commitment to good government is reflected in its core values of accountability, 

integrity, and reliability.

Obtaining Copies of 

GAO Reports and 

Testimony

The fastest and easiest way to obtain copies of GAO documents at no cost is 

through the Internet. GAO’s Web site (www.gao.gov) contains abstracts and fulltext files of current reports and testimony and an expanding archive of older 

products. The Web site features a search engine to help you locate documents 

using key words and phrases. You can print these documents in their entirety, 

including charts and other graphics.

Each day, GAO issues a list of newly released reports, testimony, and 

correspondence. GAO posts this list, known as “Today’s Reports,” on its Web site 

daily. The list contains links to the full-text document files. To have GAO e-mail this 

list to you every afternoon, go to www.gao.gov and select “Subscribe to GAO 

Mailing Lists” under “Order GAO Products” heading.

Order by Mail or Phone The first copy of each printed report is free. Additional copies are $2 each. A check 

or money order should be made out to the Superintendent of Documents. GAO 

also accepts VISA and Mastercard. Orders for 100 or more copies mailed to a single 

address are discounted 25 percent. Orders should be sent to:

U.S. General Accounting Office

441 G Street NW, Room LM

Washington, D.C. 20548

To order by Phone: Voice: (202) 512-6000 

TDD: (202) 512-2537 

Fax: (202) 512-6061

To Report Fraud, 

Waste, and Abuse in 

Federal Programs

Contact:

Web site: www.gao.gov/fraudnet/fraudnet.htm

E-mail: fraudnet@gao.gov

Automated answering system: (800) 424-5454 or (202) 512-7470

Public Affairs Jeff Nelligan, Managing Director, NelliganJ@gao.gov (202) 512-4800

U.S. General Accounting Office, 441 G Street NW, Room 7149 

Washington, D.C. 20548
United States

General Accounting Office

Washington, D.C. 20548-0001

Official Business

Penalty for Private Use $300

Address Service Requested

Presorted Standard

Postage & Fees Paid

GAO

Permit No. GI00